{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——Logistic 回归\n",
    "tfidf特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，分别调用\n",
    "缺省参数LogisticRegression、\n",
    "LogisticRegression + GridSearchCV （可用LogisticRegressionCV代替）进行参数调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.081393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.075886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.231403</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.199730</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.011987</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.011668</td>\n",
       "      <td>0.105971</td>\n",
       "      <td>0.021681</td>\n",
       "      <td>0.080435</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008244</td>\n",
       "      <td>0.022456</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.124622</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.145988</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 95 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0   1      0.081393           0.0           0.0      0.000000      0.000000   \n",
       "1   2      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "2   3      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "3   4      0.011987           0.0           0.0      0.011668      0.105971   \n",
       "4   5      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf   ...     \\\n",
       "0      0.000000      0.000000      0.000000           0.0   ...      \n",
       "1      0.000000      0.000000      0.231403           0.0   ...      \n",
       "2      0.000000      0.000000      0.199730           0.0   ...      \n",
       "3      0.021681      0.080435      0.000000           0.0   ...      \n",
       "4      0.000000      0.000000      0.000000           0.0   ...      \n",
       "\n",
       "   feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  \\\n",
       "0       0.075886       0.000000       0.000000            0.0            0.0   \n",
       "1       0.000000       0.000000       0.000000            0.0            0.0   \n",
       "2       0.000000       0.000000       0.000000            0.0            0.0   \n",
       "3       0.000000       0.008244       0.022456            0.0            0.0   \n",
       "4       0.124622       0.000000       0.000000            0.0            0.0   \n",
       "\n",
       "   feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf   target  \n",
       "0       0.000000            0.0            0.0            0.0  Class_1  \n",
       "1       0.000000            0.0            0.0            0.0  Class_1  \n",
       "2       0.000000            0.0            0.0            0.0  Class_1  \n",
       "3       0.000000            0.0            0.0            0.0  Class_1  \n",
       "4       0.145988            0.0            0.0            0.0  Class_1  \n",
       "\n",
       "[5 rows x 95 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# 请自行在log(x+1)特征和tf_idf特征上尝试，并比较不同特征的结果，\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"Otto_FE_train_tfidf.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = train['target']   \n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 默认参数的Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [ 0.64867211  0.64985632  0.64124392]\n",
      "cv logloss is: 0.646590780501\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "#数据集比较大，采用3折交叉验证\n",
    "from sklearn.model_selection import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=3, scoring='neg_log_loss')\n",
    "#%timeit loss_sparse = cross_val_score(lr, X_train_sparse, y_train, cv=3, scoring='neg_log_loss')\n",
    "print ('logloss of each fold is: ',-loss)\n",
    "print ('cv logloss is:', -loss.mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原始特征：0.797465616286\n",
    "log特征：0.684053401541\n",
    "tfidf特征：0.646590780501"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression及参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J =  C* sum(logloss(f(xi), yi)) + penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "设置参数搜索范围\n",
    "生成GridSearchCV的实例（参数）\n",
    "调用GridSearchCV的fit方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=3, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.639636895369\n",
      "{'penalty': 'l2', 'C': 100}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qing/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qing/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VPW9//HXJxtJIISYBRMiBNlk\nTZQoIIpVqyBKqHtt3bHUVq1t1att76232t/t7bX3tr3XFZG6FncEUcGlUlwAWSTsa0AIYQk7hEC2\n7++PGWKALJOQyclk3s/HIw9mzvmemc8gzjvn+z3n+zXnHCIiIgARXhcgIiKth0JBRESqKRRERKSa\nQkFERKopFEREpJpCQUREqikURESkmkJBRESqKRRERKRalNcFNFZKSorLysryugwRkZCycOHCnc65\n1IbahVwoZGVlsWDBAq/LEBEJKWb2TSDt1H0kIiLVFAoiIlItaKFgZpPMbIeZLatjf6KZvWtm+Wa2\n3MxuC1YtIiISmGCOKTwPPA68WMf+u4AVzrkxZpYKrDazV5xzZUGsSUTakPLycgoLCzl8+LDXpbQa\nsbGxZGZmEh0d3aTjgxYKzrnZZpZVXxMgwcwM6ADsBiqCVY+ItD2FhYUkJCSQlZWF76skvDnn2LVr\nF4WFhXTv3r1Jr+HlmMLjQF+gCFgK3Oucq6qtoZmNN7MFZraguLi4JWsUkVbs8OHDJCcnKxD8zIzk\n5OSTOnPyMhRGAouBDCAHeNzMOtbW0Dk3wTmX65zLTU1t8DJbEQkjCoRjnezfh5ehcBvwtvNZB2wA\nzgjWm23YWcJjM1dRXlnryYiIhInrn5nD9c/M8bqMVsvLUNgEXAxgZp2BPkBBsN5s/Y6DPPHpej5f\nuzNYbyEiYahDhw7Vj0eNGkWnTp244ooram171113kZOTQ79+/YiLiyMnJ4ecnBzefPPNRr3nokWL\nmDFjxknVXZegDTSb2WTgO0CKmRUCDwPRAM65p4FHgefNbClgwIPOuaB9Y4/onUpiXDTT8ou48Iy0\nYL2NiISxBx54gEOHDvHMM8/Uuv+JJ54AYOPGjVxxxRUsXry4Se+zaNEili1bxqhRo5pca12Cdqbg\nnLvBOZfunIt2zmU6555zzj3tDwScc0XOuUudcwOdcwOccy8HqxaAmKgILhtwKh8u30ZpWWUw30pE\nwtTFF19MQkJCk45du3YtI0eOZPDgwYwYMYI1a9YA8OqrrzJgwACys7O58MILKS0t5ZFHHuGVV15p\n0llGQ0Ju7qOTkZeTwavzN/OPVTu4fFC61+WISDP63bvLWVG0v8F2K7b62gQyrtAvoyMPj+l/0rUF\nYvz48UycOJEePXrwxRdfcPfdd/Phhx/yu9/9jlmzZtG5c2f27t1LXFwcv/3tb1m2bBl/+ctfmr2O\nsAqFId2TSUtox9TFWxQKItJq7N27l7lz53L11VdXb6uo8N22NXz4cG6++WauvfZarrrqqqDXElah\nEBlhXDEog5fnfsO+0nIS45p2x5+ItD6B/kZ/9AzhtR8PC2Y5jeKcIyUlpdYxhmeffZZ58+Yxffp0\nsrOzWbJkSVBrCbsJ8cbmZFBWWcXM5du8LkVEBICkpCTS09OZMmUKAFVVVeTn5wNQUFDA0KFDefTR\nR0lKSmLLli0kJCRw4MCBoNQSdqEwKDORbsnxTFtc5HUpItLGnH/++Vx77bV88sknZGZmMnPmzICP\nffXVV3n66afJzs6mf//+TJ8+HYBf/OIXDBw4kIEDB/Ld736XAQMGcNFFF5Gfn8+ZZ56pgeaTZWbk\nZWfwxKfr2HHgMGkJsV6XJCIh7ODBg9WPP/vss4COycrKYtmyYyeQPv3002sNkWnTpp2wLTU1NWiL\njYXdmQL4upCqHLy/ZKvXpYhIC3vtx8Na1XhCaxOWodAzLYG+6R2Zmq8uJBGRmsIyFADysjP4etNe\nNu8+5HUpIiKtRtiGwphs330K03S2ICJSLWxDITMpntxuSboKSUSkhrANBfBNe7F6+wFWbwvO9b4i\n0gr97XLfj9QqrENh9MB0IiOMaflbvC5FRELU0amzFy9ezLBhw+jfvz+DBg3itddeO6FtWE+dHQpS\nOrRjeM8UpuUXcf+lfbSCk4g0WXx8PC+++CK9evWiqKiIwYMHM3LkSDp16lTdJqynzg4VedkZbN5d\nyteb93pdioiEsN69e9OrVy8AMjIySEtLozFrymvq7FZiZP/O/HpKBNMWF3FW1ySvyxGRpvrgIdi2\ntOF22/wTygUyrnDqQLjsPxtdyldffUVZWRk9evQI+BhNnd1KJMRGc/EZaUxfspV/u6IfkRHqQhKR\nptu6dSs33XQTL7zwAhERgXXGaOrsViYvO4MPlm1jzvpdnNcrxetyRKQpAv2N/ugZwm3vNXsJ+/fv\n5/LLL+f3v/89Q4cODfg4TZ3dylx4Rhod2kXpKiQRabKysjKuvPLK6t/qG0NTZ7cysdGRjOx/Kh8s\n28aRCq3fLCKN9/rrrzN79myef/756ktNG3N1UWuZOtucc836gsGWm5vrgjFl7D/XFHPLpK945qbB\njOx/arO/vog0v5UrV9K3b9/GHRTE7qPWora/FzNb6JzLbehYjSn4De+RTHL7GKblFykURNqyNhwG\nzUHdR35RkRGMHpjOJyu3c/BIhdfliIh4QqFQw9icDA6XV/HRCq3fLBIqQq0LPNhO9u9DoVDDWV2T\n6NIpTjOnioSI2NhYdu3apWDwc86xa9cuYmObvsxw0MYUzGwScAWwwzk3oI423wH+AkQDO51zFwSr\nnkBERBhXZKfz3Gcb2F1SxintY7wsR0QakJmZSWFhYaOmk2jrYmNjyczMbPLxwRxofh54HHixtp1m\n1gl4EhjlnNtkZmlBrCVgY7O78Mw/C3h/6VZuHNrN63JEpB7R0dF0797d6zLalKB1HznnZgO762ny\nA+Bt59wmf/sdwaqlMfqmJ9AzrYNWZBORsOTlmEJvIMnMZpnZQjO72cNaqpkZedkZzN+4m6K9pV6X\nIyLSorwMhShgMHA5MBL4NzPrXVtDMxtvZgvMbEFL9B3mZWfgHExforMFEQkvXoZCITDDOVfinNsJ\nzAaya2vonJvgnMt1zuWmpqYGvbCslPZkZyaqC0lEwo6XoTAVON/MoswsHhgCrAzqO1YFPq/RmOwM\nlm3ZT0HxwSAWJCLSugQtFMxsMjAH6GNmhWY2zszuNLM7AZxzK4EZwBLgK2Cic25ZsOphzUz4azYc\nDGw8e0x2BmbobEFEwkrQLkl1zt0QQJvHgMeCVcMxTukB+7fAnCfgkt812Lxzx1iGdk9m2uIi7r24\nl9ZvFpGwED53NKf0hP5XwvyJcKi+K2W/lZeTQcHOEpYX7Q9ycSIirUP4hALA+fdB2UH4akJAzS8b\ncCrRkaYuJBEJG+EVCp37Q5/LYe5TcKThVYs6xcdwQe9U3s0voqpKc6uISNsXXqEAMOI+OLwX5j8X\nUPMx2Rls3XeY+RsD63ISEQll4RcKXQZDj4tgzuNQ3vAdy5f060xcdKS6kEQkLIRfKACcfz+UFMOi\nWufqO0Z8TBSX9OvM+0u3Ul5Z1QLFiYh4JzxDIWs4dB0GX/wVKsoabJ6XncGeQ+V8vnZnCxQnIuKd\n8AwFgBH3++5byJ/ccNPeqSTGRasLSUTavPANhR4XQ3oOfP5nqKx/TeaYqAhGDzyVmcu3UVoW+FQZ\nIiKhJnxDwQxGPAB7NsDyKQ02H5OdwaGySj5Ztb0FihMR8Ub4hgJAn9GQ2hc++xNU1T+IPKR7MmkJ\n7bR+s4i0aeEdChERvrGF4lWw+r16m0ZGGFcMymDW6mL2lZa3UIEiIi0rvEMBfPMhnXI6zH4MXP13\nLY/NyaCssoqZy7a1UHEiIi1LoRARCef9Erbmw7pP6m06KDORbsnxugpJRNoshQLAoOuhY2aDZwtH\n12/+cv1Odhw43IIFioi0DIUCQFQMDL8XNs+Fb76ot+nYnAyqHLy3ZGsLFSci0nIUCkeddRO0T4PZ\nf6q3Wc+0BPqmd1QXkoi0SQqFo6Lj4Ny7oeBTKFxYb9O87Ay+3rSXTbsOtVBxIiItQ6FQU+7tEJfk\nu2+hHmOy0wF4d4nOFkSkbVEo1NQuAYb8BFa/D9uW1dksMyme3G5JupFNRNochcLxhoyHmAT47L/r\nbZaXk8Hq7QdYtU3rN4tI26FQOF5cEpxzh28+pJ1r62w2emA6kRGmswURaVMUCrUZehdExfpmUK1D\nSod2DO+ZwrtLinAN3AktIhIqFAq16ZAKg2+F/Fdhzzd1NsvLzmDz7lK+3ry35WoTEQkihUJdzr0H\nLMK3OlsdRvbvTExUhLqQRKTNUCjUJbELnPlD+Ppl2F/73csJsdFcfEYa05dspULrN4tIGxC0UDCz\nSWa2w8zqvrbT1+5sM6s0s2uCVUuTDf85VFXAnMfrbJKXncHOg0eYW7C7BQsTEQmOYJ4pPA+Mqq+B\nmUUCfwRmBrGOpjulOwy8FhZMgpJdtTa58Iw0OrSLYuriLS1cnIhI8wtaKDjnZgMN/fp8D/AWsCNY\ndZy0838J5aUw98lad8dGRzKy/6nMWL6NIxVav1lEQptnYwpm1gW4Eng6gLbjzWyBmS0oLi4OfnE1\npfaBvmPgqwlQWvtVRnk5GRw4XMGs1S1cm4hIM/NyoPkvwIPOuQZ/vXbOTXDO5TrnclNTU1ugtOOM\nuB+O7If5z9a6e3iPZJLbx2jmVBEJeV6GQi7wqpltBK4BnjSz73lYT93Ss6HXpTDnSSgrOWF3VGQE\nlw9K5+MV2zl4pMKDAkVEmodnoeCc6+6cy3LOZQFvAj91zr3jVT0NGvEAlO6Ghc/XujsvO4MjFVV8\ntELrN4tI6ArmJamTgTlAHzMrNLNxZnanmd0ZrPcMqtPOgazz4Yv/hfITl+I8q2sSXTrF6UY2EQlp\nUcF6YefcDY1oe2uw6mhWIx6AF/Ng8Stw9rhjdkVEGGOyM5j4WQG7S8o4pX2MR0WKiDSd7mhujO4j\nIPNs+PwvUFl+wu687AwqqhzvL9X6zSISmhQKjWEG598P+zbB0jdO2N03PYGeaR10FZKIhCyFQmP1\nHgmdB/oW4ak69mpaMyMvO4OvNuymaG+pRwWKiDSdQqGxzGDEfbBrHayYesLuvOwMAKZr/WYRCUEK\nhabomwcpvX1nC8ctsJOV0p7szER1IYlISFIoNEVEJJz3S9i+DNbMOGH3mOwMlm3Zz/rigx4UJyLS\ndAqFphp4DXTqBrP/dMLZwpjsDMzQPQsiEnIUCk0VGQ3n/Ry2LICCWcfs6twxlqHdk3k3X+s3i0ho\nUSicjJwfQkK6b2zhOHk5GRTsLGHZlv0eFCYi0jQKhZMR1Q7O/Rls/Aw2zT1m12UDTiU60piWr8V3\nRCR0KBRO1uBbID7ZN7ZQQ6f4GC7oncr0JVupqlIXkoiEBoXCyYppD8PugnUfQdHXx+wak53B1n2H\nmb9R6zeLSGhodCiYWYSZdQxGMSHr7DugXeIJYwuX9OtMXHQkU3XPgoiEiIBCwcz+bmYdzaw9sAJY\nbWYPBLe0EBKbCEN+DCvfhR0rqzfHx0RxSb/OfLB0K+WVVR4WKCISmEDPFPo55/YD3wPeB7oCNwWt\nqlA09CcQ3R4++59jNudlZ7DnUDmfr93pUWEiIoELNBSizSwaXyhMdc6VAxo9rSn+FDj7dlj2Juwu\nqN48oncqiXHRTF2sq5BEpPULNBSeATYC7YHZZtYN0AX4xxt2N0REw+d/rt4UExXB6IGn8uGK7ZSW\nVdZzsIiI9wIKBefc/zrnujjnRjufb4ALg1xb6Ek4Fc66CRZPhn2F1ZvHZGdwqKyST1Zt97A4EZGG\nBTrQfK9/oNnM7DkzWwRcFOTaQtPwewHnW8vZb0j3ZNIS2jFVcyGJSCsXaPfR7f6B5kuBVOA24D+D\nVlUo69QVBn0fFr0AB3cAEOlfv/mfq4vZV3riMp4iIq1FoKFg/j9HA39zzuXX2CbHO+8XUFkGcx6v\n3pSXnUFZZRUzl23zsDARkfoFGgoLzexDfKEw08wSAF14X5eUntD/Spj/HBzy3c08KDORbsnxTNVc\nSCLSigUaCuOAh4CznXOHgBh8XUhSl/Pvg7KD8NUEwLd+89jsDOas38WOA4c9Lk5EpHaBXn1UBWQC\n/2pmfwLOdc4tCWploa5zf+hzOcx9Co4cAHzTaVc5eG/JVo+LExGpXaBXH/0ncC++KS5WAD8zsz8E\ns7A2YcR9cHivrxsJ6JmWQN/0jroKSURarUC7j0YDlzjnJjnnJgGjgMvrO8DMJpnZDjNbVsf+H5rZ\nEv/Pl2aW3bjSQ0CXwdDjIt+Ac3kp4BtwXrx5L5t2HfK4OBGREzVmltRONR4nBtD+eXzhUZcNwAXO\nuUHAo8CERtQSOs6/H0qKYdGLAIzJTgfg3SU6WxCR1ifQUPgD8LWZPW9mLwALgf+o7wDn3GygzoUE\nnHNfOuf2+J/OxTdm0fZkDYeu58IXf4WKMjKT4sntlsQ0dSGJSCsU6EDzZGAo8Lb/Z5hz7tVmrGMc\n8EEzvl7rMuI+2L8F8icDvgHn1dsPsGqbpo8Skdal3lAws7OO/gDpQCGwGcjwbztpZnYhvlB4sJ42\n481sgZktKC4ubo63bVk9LoaMM30T5VVWMHpgOpERprMFEWl1ohrY/9/17HOc5PxHZjYImAhc5pzb\nVecbOTcB/5hDbm5u6E3ZbeYbW3jth7D8bVIGXcfwnilMyy/igZF9MNPN4SLSOtQbCs65oM2EamZd\n8XVF3eScWxOs92k1+oyG1L6+JTsHXENedgb3v5HPok17GdwtyevqRESAhs8UADCzq2rZvA9Y6pzb\nUccxk4HvAClmVgg8DEQDOOeeBn4LJANP+n9TrnDO5Tb2A4SMiAgYcT+8NQ5WTWdk/8v49ZQI3s0v\nUiiISKthzjXcG2Nm7wHDgE/9m76D74qh3sAjzrmXglXg8XJzc92CBQta6u2aV1UlPJ4L7RJg/D/5\nySuLmL9xD3N/dRFRkY25OlhEpHHMbGEgv3gH+k1UBfR1zl3tnLsa6AccAYZQzwCxHCciEs77JWzN\nh3Ufk5edwc6DR5hTUOdwiohIiwo0FLKcczWXDdsB9HbO7Qa0QEBjDLoeOmbC7Me4sE8qHdpF6Sok\nEWk1Ag2Fz8xsupndYma3ANPwrdXcHtgbvPLaoKgYOO/nsHkesVvmMLL/qcxYvo3D5Vq/WUS8F2go\n3AX8DcgBzgReAO5yzpUE8wqlNuvMG6F9Gnz2J/JyMjhwuIJZq0Pw/gsRaXMCvaPZAZ8D/wA+Bma7\nQEaopXbRcXDuPVAwi+GxG0huH8O7+epCEhHvBTp19nXAV8A1wHXAPDO7JpiFtXm5t0NcElGf/w+X\nD0rn45XbOXikwuuqRCTMBdp99Bt8q67d4py7GTgH+LfglRUG2nWAIT+BNR9w/Wl7OVJRxYfLtX6z\niHgr0FCIOO4mtV2NOFbqMmQ8xCTQb91EunSKY5q6kETEY4F+sc8ws5lmdquZ3Qq8B7wfvLLCRFwS\nnHMHtuIdbuldzudrd7K7pMzrqkQkjAU60PwAvgnpBgHZwATnnG5aaw5D74KoWK47/DoVVY73l2r9\nZhHxTsBdQM65t5xzv3TO/cI5NyWYRYWVDqkw+FYS105heEqJbmQTEU81tJ7CATPbX8vPATPTCjHN\n5dx7sIhIHkyYyVcbd1O0t9TrikQkTNUbCs65BOdcx1p+EpxzHVuqyDYvsQvk/ICB26eRxh6ma/1m\nEfGIriBqLYb/HHOV/LrTx0xVF5KIeESh0Fqc0h0GXsvlZR9QVFTI+uKDXlckImFIodCanP9LoqqO\nMC5qhgacRcQTCoXWJLUP1i+P26M/5JPFa9D0UiLS0hQKrc359xHvDnHB3qks26ILvESkZSkUWpv0\nbMpPv4RxUe/zwaK1XlcjImFGodAKRV/4L5xiB2mX/xJVVepCEpGWo1BojU47h+KUIXy/4h3mr9e0\nFyLSchQKrVTCpQ/R2faybdZEr0sRkTCiUGilYntdyIbYfpy95UXKjhzxuhwRCRMKhdbKjL2595JB\nMes+meR1NSISJhQKrVj/C65lFVmkLH4Cqiq9LkdEwkDQQsHMJpnZDjNbVsd+M7P/NbN1ZrbEzM4K\nVi2hKiY6koVdbyetbDNlS972uhwRCQPBPFN4HhhVz/7LgF7+n/HAU0GsJWR1H3ED66oyKP3HY6A7\nnEUkyIIWCs652cDuepqMBV50PnOBTmaWHqx6QtWQ01N5OfpqEvevhjUzvC5HRNo4L8cUugCbazwv\n9G+TGiIjjOjs69js0qiY9V86WxCRoPIyFKyWbbV+45nZeDNbYGYLiouLg1xW63PFmV15qmIMUVsX\nQcEsr8sRkTbMy1AoBE6r8TwTqHW+aOfcBOdcrnMuNzU1tUWKa00GZSbyVeIodkckw+w/eV2OiLRh\nXobCNOBm/1VIQ4F9zjnN6VALM2N0TjeeKBsN33wO38zxuiQRaaOCeUnqZGAO0MfMCs1snJndaWZ3\n+pu8DxQA64BngZ8Gq5a2IC8ng79XXEhpdBJ8prMFEQmOqGC9sHPuhgb2O+CuYL1/W9MzLYGs9DTe\nLMvjpnUvQNHXkHGm12WJSBujO5pDyNicDP6463yq2iXCp/+hu5xFpNkpFELImOwMDhLPl24QrP0Q\n3rgFyku9LktE2hCFQgjp0imO3G5JPBL3AIz8A6ycDi+OhUP13SMoIhI4hUKIycvJYM32g6zqfiNc\n+zwULYbnLoU9G70uTUTaAIVCiBk9MJ3ICGPq4iLo/z24+R0oKYaJl/gGn0VEToJCIcSkdGhHh3ZR\nPDu7gOVF+6DbuTDuQ4iKhb9dDms/8rpEEQlhCoUQ9Oadw0hNaMd1T8/h09U7ILUP3PERJPeAv18P\ni17yukQRCVEKhRDUq3MCU346nG7J7bnjhQW8Mu8bSDgVbnsfTr8Apt0Ns/5Tk+eJSKMpFELUqYmx\nvH7nMEb0SuE3U5bxhw9WUhXdAX7wOuT8EGb9AabdA5XlXpcqIiFEoRDCOrSL4tmbc7lxaFee+WcB\n90z+msNVETD2CRjxL/D1SzD5Bjhy0OtSRSREKBRCXFRkBI+OHcBvRvfl/WVb+cGzc9lVUgYX/QbG\n/BXW/wOevxwO7vC6VBEJAQqFNsDM+NGI03nyB2exvGg/Vz75JQXFB2HwrXDDZNi5BiZ+F3au9bpU\nEWnlFAptyGUD05k8figlRyq46qkv+WrDbug9Em6dDmUl8NwlsGme12WKSCumUGhjzuqaxJSfDueU\n9jHcOHEeUxdvgS6DfZesxp0CL+bByne9LlNEWimFQhvUNTmet39yLjldO3Hvq4t5/B9rcUndfTe5\ndR4Ar90E8yZ4XaaItEIKhTaqU3wML407h+/lZPCnD9fw4FtLKI89BW55F/pcBh88AB/9FqqqvC5V\nRFoRhUIb1i4qkj9fn8PPLurJ6wsKue1v89lfFQ3Xvwy54+CLv8LbP4KKI16XKiKthEKhjTMzfnlp\nHx67ZhBzC3ZxzVNfUrjvCFz+33Dxw7DsTXj5aijd63WpItIKKBTCxLW5p/HC7eewdd9hrnzyS5Zu\n2Q/n/xKunACb5sLfLoN9hV6XKSIeUyiEkeE9U3jrJ+cSExnBdc/M4eMV2yH7erjxTV8gTLwEti/3\nukwR8ZBCIcz07pzAlLvOpVfnDox/aQEvfLkRTv8O3PYB4GDSKCj4p7dFiohnFAphKC0hllfHD+Wi\nMzrz8LTlPPLuCirT+sMdH0PHLr4xhiVveF2miHhAoRCm4mOieOamwdw2PItJX2zgJy8vpDQuHW6f\nAacNgbfvgM//rOm3RcKMQiGMRUYYD4/pz8Nj+vHRyu18f8Iciivi4Ka3YcDV8PG/w/sPQFWl16WK\nSAtRKAi3De/OhJtyWbP9IFc++QVrd5XBVRPh3Htg/rPw+s1QXup1mSLSAhQKAsAl/Trz2o+Hcri8\nique+pIvC3bDpb+HUX+EVe/BC3lQssvrMkUkyIIaCmY2ysxWm9k6M3uolv1dzexTM/vazJaY2ehg\n1iP1G5TZiXfuOpf0xFhunvQVby4shKF3wnUvwNZ83yyruzd4XaaIBFHQQsHMIoEngMuAfsANZtbv\nuGb/CrzunDsT+D7wZLDqkcBkJsXzxp3nck73U7j/jXz+/NEaXN88uGUalO72BcOWRV6XKSJBEswz\nhXOAdc65AudcGfAqMPa4Ng7o6H+cCBQFsR4JUGJcNM/fdg7XDM7kr5+s5b7X8ynLOAdu/xCi43wr\nua350OsyRSQIghkKXYDNNZ4X+rfV9O/AjWZWCLwP3BPEeqQRYqIieOyaQdx3SW/e/noLN0+ax772\n3WHcx5DSCyZ/Hxa+4HWZItLMghkKVsu24y96vwF43jmXCYwGXjKzE2oys/FmtsDMFhQXFwehVKmN\nmXHPxb34y/U5LPpmL1c99QWbyxPg1vd8d0G/+zP49D90L4NIGxLMUCgETqvxPJMTu4fGAa8DOOfm\nALFAyvEv5Jyb4JzLdc7lpqamBqlcqcv3zuzCi+POYefBMq588gu+3l4BP3gNcm6Ef/4Rpt4NleVe\nlykizSCYoTAf6GVm3c0sBt9A8rTj2mwCLgYws774QkGnAq3Q0NOTefun5xIfE8X3J8xlxsqdMPZx\nuOAhWPwy/P16OHLA6zJF5CQFLRSccxXA3cBMYCW+q4yWm9kjZpbnb3Yf8CMzywcmA7c6p76I1qpH\nagem/PRc+mV05CevLGLi5xtw33kI8v4PCmbB30bDgW1elykiJ8FC7Ts4NzfXLViwwOsywtrh8kp+\n8dpiPli2jZuHdeO3V/Qjav3H8MYtEJ8CN74Fqb29LlNEajCzhc653Iba6Y5mabTY6Eie+MFZ/HjE\n6bw45xt+/NJCSrpd5BuArij13cvwzRyvyxSRJlAoSJNERBi/Gt2XR783gE9X7+C6Z+awPaEfjPsI\n4pPhxbGwYqrXZYpIIykU5KTcNLQbz91yNht2lnDlE1+wqizZFwzp2fD6LTD3aa9LFJFGUCjISbvw\njDRe//EwKp3jmqfmMHtLlW9ajDMuhxkPwszfQFWV12WKSAAUCtIsBnRJ5J27hpOZFMdtz8/n1a+L\n4boX4ewfwZzH4a1xUHHE6zL6yri3AAALfklEQVRFpAEKBWk26YlxvHHnMIb3TOGht5fyXx+upWrU\nf8F3fwfL34aXroLSPV6XKSL1UChIs0qIjea5W3K54ZyuPDlrPT97bTGHh9zjW7Rn8zyYdBns3dzw\nC4mIJxQK0uyiIyP4jysH8NBlZzB9yVZunDiPPT3G+pb53L/Fd8nqtqVelykitVAoSFCYGXde0IP/\nu+FMlmzZx1VPfcnGhMFw+wzAfGcMBbO8LlNEjqNQkKAak53B3+8Ywt5Dvsn0Fh5Ohzs+hk6nwctX\nQ/5rXpcoIjUoFCTocrNOYcpPh9MpPoYbnp3H9G8MbvsAug6DKePhj93hs/+B9f+AQ7u9LlckrGnu\nI2kxe0rKGP/SAuZv3MODo87gzuFdsP/N8S3zWfNy1cSukJEN6Tm+n4wcaH/CjOoi0giBzn0U1RLF\niAAktY/hpXFDeODNJfxxxio27S7hkZ+vIDoywneGsG0JFC2GrYthaz6sfPfbgzt2+TYg0nN8d0wn\ndPbuw4i0UQoFaVGx0ZH89focup4SxxOfrqdwTyl/ujabtIQk7PTv+FZ0O6p0ry8otuZ/Gxar3/t2\nf0L6twFxNCwSTgWrbdE/EQmEuo/EM6/N38SDb/kuTU1oF0W3lHiyktvTPaU9WcntyUrxPU6Kj8aO\nftEf3u+7nPXo2UTRYti5huqVXtunHXs2kZHjO8tQUEiYC7T7SKEgnlpRtJ95G3axcWcJG3YdYuPO\nEgr3HKKqxj/LjrFRvqDwh8XRx92T25MYHw1HDsL2Zcd2PRWvAuefbyk+5dizifRs6NRVQSFhRaEg\nIausoorNe3wBsWFnCRt3lbBx5yE27CyhaF8pNf/JJsVHVwdEVo2wyEqEhL2rj+162rESXKXvwLhT\nfOFQMyySshQU0mYpFKRNOlxeyebdh6rDYsNOX3hs3FXC1n2Hj2mb0iHmmG6o0ztF0ce+oUvpatoV\nL/WFxY6VUFXuOyA20R8UR7uezoSk7hChK7cl9CkUJOyUllXyze4S/xmG/0xjl+/5jgPHztCaltCO\nrJT29EyK5sy4rfR16+lSuobEvSuI2LEcKst8Ddt1hFMHHdv1lNxTQSEhR6EgUkPJkYrqbijfGUZJ\n9RnGzoNlx7Q9rWMk5yXuJLfdZs5w6+lSupqO+1YTUekPlpgOvqCo2fWU0gsiIj34ZCKB0X0KIjW0\nbxdF/4xE+mcknrDvwOFy35iF/6xi484SVu1KYEZRGnsODQIgigp6RRRxXnwhudGbOGNnAV0K/0ZU\nla/LykXHYxYJEVHQbRhERkNkzHE/0cF9rLMXaQYKBQl7CbHRDMxMZGDmiYGx71B5dVgcHcd40v+4\n5PARTretDLQCBlZsYGBEAfEcod3alURRQTQVRLkKoiiv8di3PRgqiaDSoqm0KCosuvpx9Z8R326r\nquVxzT+r7MSvhkD7FBrT+eA4fmD/xIMDfrlaGhbsLAGge0r7Y7Zbna9a+/ZaLz+o84PW9Rp1tK/l\ndWp7v8M71lEe0Y5L/31mHe/bPBQKIvVIjI8mJ74TOad1Oma7c469NQJj484SfjxvEw4Y0v0UnIMq\n53D+tjWfV1U5Ilw5kVUVRLgy/5/lRLoKomo8j3JH/zy6z/dnZFU5UVQQ6W8T6XzPjwbQ0fCJchVE\nVX27LRp/KLkyojhEtD+sjgbW0ce+dpVB/7ut80uyGS8Ay3W+93E7TnzREwOp/gJqq7au12js9oDa\nmmO+6xfw8U2lMQURkTAQ6JiCOiFFRKRaUEPBzEaZ2WozW2dmD9XR5jozW2Fmy83s78GsR0RE6he0\nMQUziwSeAC4BCoH5ZjbNObeiRptewK+A4c65PWaWFqx6RESkYcE8UzgHWOecK3DOlQGvAmOPa/Mj\n4Ann3B4A59yOINYjIiINCGYodAE213he6N9WU2+gt5l9YWZzzWxUEOsREZEGBPOS1NquvTr+Uqco\noBfwHSAT+MzMBjjn9h7zQmbjgfEAXbt2bf5KRUQECO6ZQiFwWo3nmUBRLW2mOufKnXMbgNX4QuIY\nzrkJzrlc51xuampq0AoWEQl3wQyF+UAvM+tuZjHA94Fpx7V5B7gQwMxS8HUnFQSxJhERqUfQQsE5\nVwHcDcwEVgKvO+eWm9kjZpbnbzYT2GVmK4BPgQecc7uCVZOIiNQv5O5oNrNi4JsmHp4C7GzGcryk\nz9I6tZXP0lY+B+izHNXNOddg/3vIhcLJMLMFgdzmHQr0WVqntvJZ2srnAH2WxtI0FyIiUk2hICIi\n1cItFCZ4XUAz0mdpndrKZ2krnwP0WRolrMYURESkfuF2piAiIvUIu1Aws0fNbImZLTazD80sw+ua\nmsrMHjOzVf7PM8XMOjV8VOtkZtf6p0+vMrOQu1IkkGniQ4GZTTKzHWa2zOtaTpaZnWZmn5rZSv+/\nrXu9rqkpzCzWzL4ys3z/5/hdUN8v3LqPzKyjc26///HPgH7OuTs9LqtJzOxS4B/OuQoz+yOAc+5B\nj8tqEjPrC1QBzwD3O+dCZnk9/zTxa6gxTTxwQ81p4kOFmY0ADgIvOucGeF3PyTCzdCDdObfIzBKA\nhcD3Qu2/i5kZ0N45d9DMooHPgXudc3OD8X5hd6ZwNBD82tOIdcFbG+fch/47xwHm4ptfKiQ551Y6\n51Z7XUcTBTJNfEhwzs0GdntdR3Nwzm11zi3yPz6Ab2aF42dqbvWcz0H/02j/T9C+t8IuFADM7P+Z\n2Wbgh8Bvva6nmdwOfOB1EWEqkGnixUNmlgWcCczztpKmMbNIM1sM7AA+cs4F7XO0yVAws4/NbFkt\nP2MBnHO/cc6dBryCb36mVquhz+Jv8xugAt/nabUC+SwhKpBp4sUjZtYBeAv4+XE9BSHDOVfpnMvB\n1xtwjpkFrWsvmOspeMY5990Am/4deA94OIjlnJSGPouZ3QJcAVzsWvkAUSP+u4SaQKaJFw/4++Df\nAl5xzr3tdT0nyzm318xmAaOAoFwM0CbPFOrjXxf6qDxglVe1nCz/SnUPAnnOuUNe1xPGApkmXlqY\nf4D2OWClc+5/vK6nqcws9eiVhWYWB3yXIH5vhePVR28BffBd6fINcKdzbou3VTWNma0D2gFHpxuf\nG8JXUl0J/B+QCuwFFjvnRnpbVeDMbDTwFyASmOSc+38el9QkZjYZ30qIKcB24GHn3HOeFtVEZnYe\n8BmwFN//7wC/ds69711VjWdmg4AX8P3bisC3DMEjQXu/cAsFERGpW9h1H4mISN0UCiIiUk2hICIi\n1RQKIiJSTaEgIiLVFAoitTCzgw23qvf4N83sdP/jDmb2jJmt989yOdvMhphZjP9xm7yJVEKTQkGk\nmZlZfyDSOVfg3zQR3yRzvZxz/YFbgRT/5HmfANd7UqhILRQKIvUwn8f8czQtNbPr/dsjzOxJ/2/+\n083sfTO7xn/YD4Gp/nY9gCHAvzrnqgD8s6m+52/7jr+9SKug01aR+l0F5ADZ+O7ynW9ms4HhQBYw\nEEjDNy3zJP8xw4HJ/sf98d2dXVnH6y8Dzg5K5SJNoDMFkfqdB0z2z1K5Hfgnvi/x84A3nHNVzrlt\nwKc1jkkHigN5cX9YlPkXgRHxnEJBpH61TYtd33aAUiDW/3g5kG1m9f2/1g443ITaRJqdQkGkfrOB\n6/2LnKQCI4Cv8C2JeLV/bKEzvknkjloJ9ARwzq0HFgC/88/aiZn1OrqGhJklA8XOufKW+kAi9VEo\niNRvCrAEyAf+AfyLv7voLXzrKCzDt670PGCf/5j3ODYk7gBOBdaZ2VLgWb5db+FCIKRm7ZS2TbOk\nijSRmXXwL6aejO/sYbhzbpt/zvtP/c/rGmA++hpvA78K4fWppY3R1UciTTfdv/hJDPCo/wwC51yp\nmT2Mb53mTXUd7F+Q5x0FgrQmOlMQEZFqGlMQEZFqCgUREammUBARkWoKBRERqaZQEBGRagoFERGp\n9v8BbPUqJh+2dwkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ab05e90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    #plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'logloss' )\n",
    "plt.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上测试集上的logloss。可以看出在训练集上C越大（正则越少）的模型性能越好；但在测试集上当C=100时性能最好（L1正则）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用LogisticRegressionCV实现正则化的 Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L1正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "           class_weight=None, cv=3, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1.0, max_iter=100, multi_class='ovr',\n",
       "           n_jobs=1, penalty='l1', random_state=None, refit=True,\n",
       "           scoring='neg_log_loss', solver='liblinear', tol=0.0001,\n",
       "           verbose=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]\n",
    "#nCs = 9  #Cs values are chosen in a logarithmic scale between 1e-4 and 1e4.\n",
    "\n",
    "# 大量样本（6W+）、高维度（93），L1正则 --> 可选用saga优化求解器(0.19版本新功能)\n",
    "# LogisticRegressionCV比GridSearchCV快\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 3, scoring='neg_log_loss', penalty='l1', solver='liblinear', multi_class='ovr')\n",
    "lrcv_L1.fit(X_train, y_train)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Class_1': array([[-0.14540941, -0.13888327, -0.11033725, -0.08749023, -0.08507874,\n",
       "         -0.08521415, -0.08524412],\n",
       "        [-0.14541674, -0.13889324, -0.11085722, -0.08637617, -0.0836447 ,\n",
       "         -0.08395953, -0.08400906],\n",
       "        [-0.14542407, -0.1389033 , -0.10961539, -0.08513374, -0.08210621,\n",
       "         -0.08207292, -0.08208421]]),\n",
       " 'Class_2': array([[-0.57507308, -0.45068348, -0.34931526, -0.32030421, -0.31669393,\n",
       "         -0.31659969, -0.31659842],\n",
       "        [-0.5750947 , -0.45161968, -0.34949774, -0.32071066, -0.31833695,\n",
       "         -0.31854972, -0.31858148],\n",
       "        [-0.57511632, -0.44816622, -0.34621376, -0.31954067, -0.31724365,\n",
       "         -0.31728877, -0.31729992]]),\n",
       " 'Class_3': array([[-0.38749796, -0.34550656, -0.28295377, -0.26357786, -0.26132108,\n",
       "         -0.26135071, -0.26136374],\n",
       "        [-0.38751797, -0.34476714, -0.2853468 , -0.26844802, -0.2666547 ,\n",
       "         -0.26663302, -0.26663829],\n",
       "        [-0.38753799, -0.345958  , -0.28507455, -0.26546101, -0.26289776,\n",
       "         -0.26297127, -0.26298316]]),\n",
       " 'Class_4': array([[-0.18415472, -0.17893305, -0.1464004 , -0.13108627, -0.12861342,\n",
       "         -0.12885358, -0.12891715],\n",
       "        [-0.18416452, -0.17894572, -0.14574117, -0.12915447, -0.12653613,\n",
       "         -0.12661015, -0.12666254],\n",
       "        [-0.18417431, -0.1789584 , -0.14722002, -0.13256127, -0.13008921,\n",
       "         -0.1301892 , -0.13022441]]),\n",
       " 'Class_5': array([[-0.18648008, -0.07880459, -0.03785669, -0.01959767, -0.01458579,\n",
       "         -0.01422857, -0.01448869],\n",
       "        [-0.18649002, -0.08068992, -0.04168659, -0.02229938, -0.01630894,\n",
       "         -0.01630374, -0.01660148],\n",
       "        [-0.18649996, -0.08324412, -0.04253587, -0.02253446, -0.01632   ,\n",
       "         -0.01552271, -0.01570203]]),\n",
       " 'Class_6': array([[-0.53896138, -0.22606402, -0.13831651, -0.11811352, -0.1183933 ,\n",
       "         -0.11870177, -0.11873754],\n",
       "        [-0.53898383, -0.22334704, -0.13362228, -0.10875925, -0.10683454,\n",
       "         -0.10683099, -0.10683452],\n",
       "        [-0.53895681, -0.22146801, -0.13126193, -0.10825376, -0.10690238,\n",
       "         -0.10708653, -0.10711044]]),\n",
       " 'Class_7': array([[-0.19136209, -0.17442254, -0.1274818 , -0.10595623, -0.10419447,\n",
       "         -0.10434829, -0.10437998],\n",
       "        [-0.19125602, -0.1737928 , -0.1283954 , -0.10997782, -0.10838408,\n",
       "         -0.10855415, -0.10857447],\n",
       "        [-0.19126625, -0.17360164, -0.12434223, -0.10261606, -0.09992786,\n",
       "         -0.09999191, -0.10001154]]),\n",
       " 'Class_8': array([[-0.40138531, -0.30024447, -0.136595  , -0.10852201, -0.10712896,\n",
       "         -0.10736753, -0.10739802],\n",
       "        [-0.40133013, -0.30120056, -0.13294125, -0.10269237, -0.10081911,\n",
       "         -0.1009001 , -0.10091663],\n",
       "        [-0.40135062, -0.29938001, -0.13476343, -0.10676062, -0.1048938 ,\n",
       "         -0.10500785, -0.10502702]]),\n",
       " 'Class_9': array([[-0.28236401, -0.2212519 , -0.11548552, -0.09069683, -0.08833885,\n",
       "         -0.08879295, -0.08887426],\n",
       "        [-0.2823794 , -0.22325211, -0.11742408, -0.09472103, -0.09206397,\n",
       "         -0.09216805, -0.09220315],\n",
       "        [-0.28229682, -0.22399186, -0.11796379, -0.09341746, -0.09053635,\n",
       "         -0.09056858, -0.09058582]])}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4FeX5//H3nZ0dgYCssi8REGpw\nQ3FXcEFcWrW1ta39Uq3+bGtVRG296r50/9a22lrbfqtSFVFaQdwAV5QgyBaWACIBkaDsS0LI/fvj\nDPYIgZzkZDInyed1XefyzDPPzLmnFT7O9jzm7oiIiNRUWtQFiIhI/aYgERGRpChIREQkKQoSERFJ\nioJERESSoiAREZGkKEhERCQpChIREUmKgkRERJKSEXUBdaFdu3bevXv3qMsQEalX5syZs9Hdc6vq\n1yiCpHv37hQUFERdhohIvWJmqxPpp0tbIiKSlFCDxMxGmtlSMysys1sqWX+1mS0ws3lm9paZ5QXt\nZ5rZnGDdHDM7LW6bGcE+5wWf9mEeg4iIHFpol7bMLB14GDgTKAZmm9lkd18c1+1Jd/9T0H808Ctg\nJLARON/d15nZQGAa0Dluu2+4u65ViYikgDDPSI4Bitx9pbuXAROAC+I7uPvWuMVmgAftc919XdC+\nCMgxs+wQaxURkRoKM0g6A2vilov58lkFAGZ2rZmtAB4Erq9kPxcDc929NK7t8eCy1k/NzCr7cTMb\na2YFZlZQUlJS86MQEZFDCjNIKvsL/oBZtNz9YXfvBYwDbv/SDsyOBB4Avh/X/A13HwScFHy+WdmP\nu/uj7p7v7vm5uVU+vSYiIjUUZpAUA13jlrsA6w7SF2KXvsbsWzCzLsAk4FvuvmJfu7uvDf65DXiS\n2CU0ERGJSJhBMhvoY2Y9zCwLuAyYHN/BzPrELZ4LLA/aWwMvAuPd/e24/hlm1i74ngmcBywM6wBe\nWrieJ9/7OKzdi4g0CKE9teXu5WZ2HbEnrtKBv7r7IjO7Eyhw98nAdWZ2BrAH2ARcGWx+HdAb+KmZ\n/TRoOwvYAUwLQiQdeBX4c1jH8PzctbyxvIQz8trTvkVOWD8jIlKvmfsBty0anPz8fK/Jm+2rNu7g\nzF/N5GvDunLvhYNCqExEJHWZ2Rx3z6+qn95sP4Qe7ZpxxXFH8K/Za1j+6baoyxERSUkKkipcf3of\nmmamc//UJVGXIiKSkhQkVWjTLIsfnNqb15Zs4J0VG6MuR0Qk5ShIEvCd4d3p1CqHe6cUUlHR8O8p\niYhUh4IkATmZ6dw0sh8L125l8oeHehVGRKTxUZAk6IKjOjOwc0semraU3Xv2Rl2OiEjKUJAkKC3N\nuHXUANZu3sXf3vko6nJERFKGgqQaTujdjtP6t+fh6UV8vqMs6nJERFKCgqSaxo/qz47Scn732vKo\nSxERSQkKkmrq06EFlw7rxj9nrWbVxh1RlyMiEjkFSQ38+Mw+ZGWk8eBLeklRRERBUgPtW+Tw/RG9\nmLpwPXNWfx51OSIikVKQ1ND/jOhB+xbZ3PNiIY1h4EsRkYNRkNRQ06wMfnJWXz74eDNTF66PuhwR\nkcgoSJJwydFd6dehBQ+8tISy8oqoyxERiYSCJAnpacb4c/qz+rOd/HPW6qjLERGJhIIkSSf3zeXE\n3u343evL2bJrT9TliIjUuVCDxMxGmtlSMysys1sqWX+1mS0ws3lm9paZ5cWtGx9st9TMzk50n3XN\nLHZWsmXXHv4wvSjqckRE6lxoQWJm6cDDwCggD7g8PigCT7r7IHcfAjwI/CrYNg+4DDgSGAn8wczS\nE9xnnTuyUysuGtqFx9/5iDWf74y6HBGROhXmGckxQJG7r3T3MmACcEF8B3ffGrfYDNj3HO0FwAR3\nL3X3VUBRsL8q9xmVG8/uiwG/eHlp1KWIiNSpMIOkM7Ambrk4aPsSM7vWzFYQOyO5voptE9pnFDq2\nasL3TurBC/PWMb94c9TliIjUmTCDxCppO+DNPXd/2N17AeOA26vYNqF9ApjZWDMrMLOCkpKSBEtO\nztUn96Jtsyy9pCgijUqYQVIMdI1b7gIcanrBCcCYKrZNeJ/u/qi757t7fm5ubjVLr5kWOZn86Iw+\nvLfqc14r3FAnvykiErUwg2Q20MfMephZFrGb55PjO5hZn7jFc4F9Y7NPBi4zs2wz6wH0Ad5PZJ9R\nu+yYbvRs14z7phZSvlcvKYpIwxdakLh7OXAdMA0oBJ5290VmdqeZjQ66XWdmi8xsHnADcGWw7SLg\naWAx8BJwrbvvPdg+wzqGmshMT+OWUf1ZUbKDCbPXVL2BiEg9Z43hWn5+fr4XFBTU2e+5O5c+MouV\nG7cz46ZTaZ6dUWe/LSJSW8xsjrvnV9VPb7aHwMy49dwBbNxexiMzV0RdjohIqBQkIRnStTXnH9WJ\nP7+5kvVbdkddjohIaBQkIbr57H5UVMAv9ZKiiDRgCpIQdW3TlCtPOIJnPyim8JOtVW8gIlIPKUhC\ndt2pfWiZk8m9UwqjLkVEJBQKkpC1aprJ/zutN28u38jMZXXzhr2ISF1SkNSBbx5/BF3bNOG+KYXs\nrWj4j1uLSOOiIKkD2RnpjBvZnyXrtzHxg+KoyxERqVUKkjpy7qCODOnaml++vJSdZeVRlyMiUmsU\nJHXEzLjt3AF8urWUx95cFXU5IiK1RkFSh4Z1b8PZR3bgTzNXULKtNOpyRERqhYKkjo0b2Z/S8gp+\n8+qyqEsREakVCpI61jO3Od84thsTZq+haMO2qMsREUmagiQC15/eh6aZ6dw/dUnUpYiIJE1BEoG2\nzbO55tRevFq4gXdXfBZ1OSIiSVGQROS7w3vQqVUO904ppEIvKYpIPaYgiUhOZjo3nt2PBWu38O/5\nh5rKXkQktSlIIjRmSGeO7NSSB19ayu49e6MuR0SkRkINEjMbaWZLzazIzG6pZP0NZrbYzOab2Wtm\ndkTQfqqZzYv77DazMcG6v5nZqrh1Q8I8hjClpRm3nTOAtZt38fd3Poq6HBGRGgktSMwsHXgYGAXk\nAZebWd5+3eYC+e4+GHgWeBDA3ae7+xB3HwKcBuwEXo7b7qZ96919XljHUBdO6N2OU/vl8vvpRWza\nURZ1OSIi1RbmGckxQJG7r3T3MmACcEF8hyAwdgaLs4AuleznEmBqXL8GZ/w5A9hRWs7vXl8edSki\nItUWZpB0BtbELRcHbQdzFTC1kvbLgKf2a7snuBz2azPLTq7M6PXt0IJLh3Xln7NW89HGHVGXIyJS\nLWEGiVXSVulzrmZ2BZAPPLRfe0dgEDAtrnk80B8YBrQBxh1kn2PNrMDMCkpKUn9CqR+f0ZfM9DQe\nnKaXFEWkfgkzSIqBrnHLXYADnnM1szOA24DR7r7/SIZfAya5+559De7+iceUAo8Tu4R2AHd/1N3z\n3T0/Nzc3yUMJX/uWOYwd0ZMpC9YzZ/WmqMsREUlYmEEyG+hjZj3MLIvYJarJ8R3MbCjwCLEQ2VDJ\nPi5nv8tawVkKZmbAGGBhCLVH4n9O6klui2zueXEx7npJUUTqh9CCxN3LgeuIXZYqBJ5290VmdqeZ\njQ66PQQ0B54JHuX9ImjMrDuxM5qZ++36CTNbACwA2gF3h3UMda1ZdgY/ObMvH3y8mZcWro+6HBGR\nhFhj+C/f/Px8LygoiLqMhOytcEb99g1Kyyt45ccnk5Whd0ZFJBpmNsfd86vqp7+lUkx6mjH+nAGs\n/mwnT7y3OupyRESqpCBJQaf0zWV477b87rXlbNm1p+oNREQipCBJQWbG+FED2LxrD3+YURR1OSIi\nh6QgSVEDO7fiwqGdefztjyje1GBf6heRBkBBksJuPKsfBvxi2tKoSxEROSgFSQrr1LoJV53Yg+fn\nrWNB8ZaoyxERqZSCJMVdfUov2jTL4p4peklRRFKTgiTFtczJ5Edn9GHWys95fUllL/+LiERLQVIP\nXH5MN3q2a8a9Uwop31sRdTkiIl+iIKkHMtPTGDeqPytKdvCvgjVVbyAiUocUJPXEWXkdGNb9MH79\nynK2l5ZHXY6IyBcUJPWEmXHrOQPYuL2UR2euiLocEZEvKEjqkaHdDuO8wR159M2VrN+yO+pyREQA\nBUm9c/PZ/dlb4fzqFb2kKCKpQUFSz3Rr25Qrj+/OM3OKKfxka9TliIgoSOqj607rTYvsDO6bqvnd\nRSR6CpJ6qHXTLK4/vQ9vLCvhjWUlUZcjIo1cqEFiZiPNbKmZFZnZLZWsv8HMFpvZfDN7zcyOiFu3\nN5h+d/8peHuY2XtmttzM/hXMB9/ofPP4I+japgn3Tilkb4WGThGR6IQWJGaWDjwMjALygMvNLG+/\nbnOBfHcfDDwLPBi3bpe7Dwk+o+PaHwB+7e59gE3AVWEdQyrLzkjn5rP7s2T9Np77oDjqckSkEQvz\njOQYoMjdV7p7GTABuCC+g7tPd/d9k23MArocaodmZsBpxEIH4O/AmFqtuh45b3BHjuraml++vIxd\nZXujLkdEGqkwg6QzED+eR3HQdjBXAVPjlnPMrMDMZpnZvrBoC2x2932vdle1zwbNzLjtnAGs37qb\nx95aGXU5ItJIhRkkVklbpRfzzewKIB94KK65m7vnA18HfmNmvaq5z7FBEBWUlDTcG9LH9GjDWXkd\n+OOMFZRsK426HBFphMIMkmKga9xyF2Dd/p3M7AzgNmC0u3/xN6G7rwv+uRKYAQwFNgKtzSzjUPsM\ntnvU3fPdPT83Nzf5o0lh40b1Z3d5Bb99bVnUpYhIIxRmkMwG+gRPWWUBlwGT4zuY2VDgEWIhsiGu\n/TAzyw6+twOGA4s9NrPTdOCSoOuVwAshHkO90Cu3Od84thtPvb+Gog3boy5HRBqZ0IIkuI9xHTAN\nKASedvdFZnanme17CushoDnwzH6P+Q4ACszsQ2LBcb+7Lw7WjQNuMLMiYvdMHgvrGOqTH57ehyaZ\n6dyvlxRFpI5ZY5i+NT8/3wsKCqIuI3QPTy/ioWlLmTD2OI7r2TbqckSknjOzOcG96kPSm+0NyFUn\n9qBjqxzunVJIhV5SFJE6oiBpQHIy07nxrH7ML97Cv+dX+gyCiEitU5A0MBcO7Uxex5Y8+NJSdu/R\nS4oiEr5qB4mZpZlZyzCKkeSlpRm3nTuAtZt38Y93P4q6HBFpBBIKEjN70sxamlkzYDGw1MxuCrc0\nqanhvdtxSr9cfv96EZt2lEVdjog0cImekeS5+1Zi41pNAboB3wytKkna+FED2F5azv++XhR1KSLS\nwCUaJJlmlkksSF5w9z0cZGgSSQ39Dm/B1/K78n+zPmL1ZzuiLkdEGrBEg+QR4COgGfBGMG+I5nlN\ncTec2ZeMtDQefEnzu4tIeBIKEnf/nbt3dvdzPGY1cGrItUmS2rfMYeyInry44BPmrN4UdTki0kAl\nerP9h8HNdjOzx8zsA2LzgkiKGzuiJ7ktsrl3SiGNYRQDEal7iV7a+m5ws/0sIBf4DnB/aFVJrWmW\nncENZ/ZlzupNTF24PupyRKQBSjRI9s0Dcg7wuLt/SOVzg0gK+urRXeh/eAt+/u9FbN29J+pyRKSB\nSTRI5pjZy8SCZJqZtQAqwitLalNGehr3XzyYDdtKefAljQ4sIrUr0SC5CrgFGBbMsZ5F7PKW1BND\nurbmOyf04J+zPqbgo8+jLkdEGpBEn9qqIDYb4e1m9gvgBHefH2plUut+clZfOrduwriJ8ykt1zhc\nIlI7En1q637gh8SGR1kMXG9m94VZmNS+ZtkZ3HPhQFaU7ODh6SuiLkdEGohEL22dA5zp7n91978C\nI4FzwytLwnJKv/aMGdKJP84oYtmn26IuR0QagOqM/ts67nur2i5E6s5Pz8ujeXYG4ybOZ68mwBKR\nJCUaJPcBc83sb2b2d2AOcG9VG5nZSDNbamZFZnZLJetvMLPFZjbfzF4Lhl7BzIaY2btmtihYd2nc\nNn8zs1XBHO/zzGxIgscggbbNs/nZ+XnM/Xgz/5y1OupyRKSeS/Rm+1PAccBzwed4d59wqG3MLB14\nGBgF5AGXm1neft3mAvnuPhh4FngwaN8JfMvdjyR2Ge03ZhZ/RnSTuw8JPvMSOQb5sjFDOjOiby4P\nvrSEdZt3RV2OiNRjhwwSM/vKvg/QESgG1gCdgrZDOQYocveV7l4GTAAuiO/g7tODx4kBZhF7Mgx3\nX+buy4Pv64ANxN6ol1piZtwzZiAVDj99fqGGTxGRGsuoYv0vD7HOOfR4W52Jhc4+xcCxh+h/FTB1\n/0YzO4bYeyvxjxndY2Y/A14DbnH30kPsVw6ia5um/OSsvtz9YiH/mf8J5x/VKeqSRKQeOmSQuHsy\nI/xWNoRKpf/Za2ZXAPnAyfu1dwT+D7gyeJcFYDywnli4PAqMA+6sZJ9jgbEA3bp1q9kRNALfGd6D\nyR+u4+f/XsRJfdrRumlW1CWJSD2T6HskF1XyOd3M2h9is2Kga9xyF2BdJfs+A7gNGB1/ZhHMC/8i\ncLu7z9rX7u6fBEPZlwKPE7uEdgB3f9Td8909PzdXV8UOJj3NuP+iwWzauYd7XiyMuhwRqYeqM0TK\nX4BvBJ8/AzcAb5vZwabcnQ30MbMeZpYFXAZMju9gZkOJTZo12t03xLVnAZOAf7j7M/tt0zH4pxGb\nsXFhgscgB5HXqSXfH9GTZ+YU89byjVGXIyL1TKJBUgEMcPeL3f1iYk9hlRK75zGusg3cvRy4DpgG\nFAJPu/siM7vTzEYH3R4CmgPPBI/y7guarwEjgG9X8pjvE2a2AFgAtAPurs4BS+WuP70PPdo149ZJ\nC9hVpuFTRCRxlsjTOma2wN0HxS0bsMDdB5rZXHcfGmaRycrPz/eCgoKoy0h57674jMv/PIvvj+jJ\n+HMGRF2OiETMzOa4e35V/RI9I3nTzP5jZlea2ZXELlG9YWbNgM3JFCqp4/hebblsWFf+8tYqFq7d\nEnU5IlJPJBok1xK7sT0EGAr8HbjW3Xck+WSXpJjxowbQplkW4ybOp3yvppwRkaol+ma7A28BrwOv\nAm+43mBrkFo1zeTO0UeyaN1WHntrVdTliEg9kOjjv18D3gcuIXYj/D0zuyTMwiQ6Iwcezpl5Hfj1\nq8tY/dmOqMsRkRSX6KWt24jNjnilu3+L2LsbPw2vLImSmXHXBQPJTEvj1kkLNHyKiBxSokGSFv+e\nB/BZNbaVeujwVjmMG9Wft4s+49k5xVGXIyIpLNEweMnMppnZt83s28TeOJ8SXlmSCr5+TDeGdT+M\nu18spGSbhjMTkcolerP9JmLjWg0GjgIedfdKX0SUhiMtzbjvokHsKtvLnf9ZHHU5IpKiqhr99wvu\nPhGYGGItkoJ6t2/Bdaf15levLGPMkE6cPqBD1CWJSIqpaj6SbWa2tZLPNjPbWldFSrSuPrkXfTs0\n5/bnF7K9tDzqckQkxRwySNy9hbu3rOTTwt1b1lWREq2sjDTuv3gw67fu5hfTlkZdjoikGD15JQn5\nSrfDuPL47vz93Y+Ys3pT1OWISApRkEjCbjy7Hx1b5jD+ufmUlWv4FBGJUZBIwppnZ3D3hQNZ9ul2\n/jRzRdUbiEijoCCRajmtfwfOP6oTv3+9iKIN26IuR0RSgIJEqu2O8/Nomp3OLRMXUFGh4VNEGjsF\niVRbu+bZ3H5uHgWrN/HE+x9HXY6IRExBIjVy8Vc6c2LvdjwwdQnrt+yOuhwRiVCoQWJmI81sqZkV\nmdktlay/wcwWm9l8M3vNzI6IW3elmS0PPlfGtR9tZguCff4umPZX6piZcc+FAymvqOD25xdqhGCR\nRiy0IDGzdOBhYBSQB1xuZnn7dZsL5Lv7YOBZ4MFg2zbAHcCxxIasv8PMDgu2+SMwFugTfEaGdQxy\naEe0bcYNZ/bl1cJPmbpwfdTliEhEwjwjOQYocveV7l4GTAAuiO/g7tPdfWewOAvoEnw/G3jF3T93\n903AK8BIM+sItHT3d4MZGv8BjAnxGKQK3x3eg4GdW3LH5EVs2bkn6nJEJAJhBklnYE3ccnHQdjBX\nAVOr2LZz8L3KfZrZWDMrMLOCkpKSapYuicpIT+P+iwbz+Y4y7ptaGHU5IhKBMIOksnsXlV5IN7Mr\ngHzgoSq2TXif7v6ou+e7e35ubm4C5UpNDezciu+d1IMJs9fwzoqNUZcjInUszCApBrrGLXcB1u3f\nyczOIDaV72h3L61i22L+e/nroPuUuvej0/tyRNum3PrcAnbv2Rt1OSJSh8IMktlAHzPrYWZZwGXA\n5PgOZjYUeIRYiMRP5TsNOMvMDgtusp8FTHP3T4BtZnZc8LTWt4AXQjwGSVCTrHTuvXAQH322k9+9\ntjzqckSkDoUWJO5eDlxHLBQKgafdfZGZ3Wlmo4NuDwHNgWfMbJ6ZTQ62/Ry4i1gYzQbuDNoArgH+\nAhQBK/jvfRWJ2PDe7fjq0V145I2VLF6n6WpEGgtrDM//5+fne0FBQdRlNAqbd5Zxxq9m0ql1Eyb9\nYDjpaXrNR6S+MrM57p5fVT+92S61qnXTLO44/0jmF2/h8bdXRV2OiNQBBYnUuvMGd+T0/u355cvL\nWPP5zqo3EJF6TUEitc7MuGvMQNIMbp20QMOniDRwChIJRafWTRg3qj9vLt/IpLlroy5HREKkIJHQ\nXHHsEXylW2vu+s9iPtteWvUGIlIvKUgkNGlpxv0XD2Z7aTl3/Wdx1OWISEgUJBKqvh1a8INTevP8\nvHVMX7qh6g1EpN5RkEjofnBqL3q3b87tkxayo7Q86nJEpJYpSCR02Rnp3H/RINZu3sUvX14WdTki\nUssUJFIn8ru34ZvHHcHj76xi7seboi5HRGqRgkTqzM0j+9GhRQ7jn1vAnr0VUZcjIrVEQSJ1pkVO\nJneNGciS9dt49I2VUZcjIrVEQSJ16sy8Dpw7qCO/fW05K0q2R12OiNQCBYnUuTtG55GTkcb45xZQ\nUaHhU0TqOwWJ1Ln2LXK4/dw83l/1Of8qWBN1OSKSJAWJROKr+V04vmdb7p1SyKdbd0ddjogkQUEi\nkTAz7r1oEGXlFdzxwqKoyxGRJIQaJGY20syWmlmRmd1SyfoRZvaBmZWb2SVx7acGU+/u++w2szHB\nur+Z2aq4dUPCPAYJT492zfjRGX15adF6Xlq4PupyRKSGQgsSM0sHHgZGAXnA5WaWt1+3j4FvA0/G\nN7r7dHcf4u5DgNOAncDLcV1u2rfe3eeFdQwSvu+d1IMBHVvysxcWsmXXnqjLEZEaCPOM5BigyN1X\nunsZMAG4IL6Du3/k7vOBQ72ddgkw1d011V4DlJmexgMXD2Lj9lIeeGlJ1OWISA2EGSSdgfhHcoqD\ntuq6DHhqv7Z7zGy+mf3azLJrWqCkhsFdWnPViT148r2PeW/lZ1GXIyLVFGaQWCVt1XppwMw6AoOA\naXHN44H+wDCgDTDuINuONbMCMysoKSmpzs9KBH58Zl+6tmnC+EkL2L1nb9TliEg1hBkkxUDXuOUu\nwLpq7uNrwCR3/+Liubt/4jGlwOPELqEdwN0fdfd8d8/Pzc2t5s9KXWualcG9Fw5iZckOHp5eFHU5\nIlINYQbJbKCPmfUwsyxil6gmV3Mfl7PfZa3gLAUzM2AMsLAWapUUcFKfXC76Smf+OGMFS9Zvjboc\nEUlQaEHi7uXAdcQuSxUCT7v7IjO708xGA5jZMDMrBr4KPGJmX7xQYGbdiZ3RzNxv10+Y2QJgAdAO\nuDusY5C699Nz82jVJJNbJi5gr4ZPEakXzL3h/2HNz8/3goKCqMuQBL0wby0/nDCPO87P4zvDe0Rd\njkijZWZz3D2/qn56s11SzuijOnFKv1wemraU4k166lsk1SlIJOWYGXePGQjA7c8vpDGcNYvUZwoS\nSUldDmvKTWf3Y8bSEiZ/WN2H/USkLilIJGV96/juDOnamp//ezGf7yiLuhwROQgFiaSs9DTj/osH\nsXXXHu5+cXHU5YjIQShIJKX1P7wl15zSi+c+WMubyzVCgUgqUpBIyrv21N70zG3GrZMWsLOsPOpy\nRGQ/ChJJeTmZ6dx34SDWfL6LX7+yLOpyRGQ/ChKpF47t2ZavH9uNx95axfzizVGXIyJxFCRSb9wy\nqj/tmmczbuICtpfqEpdIqlCQSL3RMieTu8cMpPCTrZxw32v8YtpSPtteGnVZIo2egkTqlbOOPJwX\nrh3O8N7teHhGEcMfeJ07XljIms81lIpIVDRoo9RbK0q288jMFUyau5YKj43R9f2Te9L/8JZRlybS\nICQ6aKOCROq9T7bs4rE3V/Hk+x+zs2wvp/dvzzWn9CK/e5uoSxOp1xQkcRQkjcPmnWX8493VPP72\nKjbt3MOw7odxzSm9OLVfe2LzoIlIdShI4ihIGpedZeU8PXsNf35zFWs376L/4S24+uRenDe4Ixnp\nui0okigFSRwFSeO0Z28F//5wHX+auYJln26ny2FNGDuiJ189uitNstKjLk8k5SlI4ihIGreKCuf1\nJRv4w4wiPvh4M22bZfGd4d355nHdadU0M+ryRFJWSsyQaGYjzWypmRWZ2S2VrB9hZh+YWbmZXbLf\nur1mNi/4TI5r72Fm75nZcjP7l5llhXkMUv+lpRln5HVg4jUn8PT3j2dwl1b84uVlDH/gde6bUsin\nW3dHXaJIvRbaGYmZpQPLgDOBYmA2cLm7L47r0x1oCdwITHb3Z+PWbXf35pXs92ngOXefYGZ/Aj50\n9z8eqhadkcj+Fq/byiNvrODfH64jIy2Ni4/uzNgRvejRrlnUpYmkjFQ4IzkGKHL3le5eBkwALojv\n4O4fuft8oCKRHVrs0ZvTgH2B83dgTO2VLI1FXqeW/Payocy48VQuHdaViR+s5bRfzuDaJz5gQfGW\nqMsTqVfCDJLOwJq45eKgLVE5ZlZgZrPMbF9YtAU2u/u+gZYOuk8zGxtsX1BSonkspHLd2jblrjED\neXvcaVxzci/eWFbC+b9/i28+9h7vFG3UfPEiCQgzSCp7cL86fyq7BadUXwd+Y2a9qrNPd3/U3fPd\nPT83N7caPyuNUW6LbG4e2Z+3x5/GLaP6s2T9Nr7+l/cY8/DbvLTwEyoqFCgiBxNmkBQDXeOWuwDr\nEt3Y3dcF/1wJzACGAhuB1maWUZN9ilSlZU4mV5/cizdvPpV7LxzE5l17uPqfH3DGr2fy9Ow1lJUn\ndBVWpFEJM0hmA32Cp6yygMvoabsQAAAJCElEQVSAyVVsA4CZHWZm2cH3dsBwYLHHrjNMB/Y94XUl\n8EKtVy6NXk5mOl8/thuv/+QUfv/1oTTJTOfmifMZ8eB0/vLmSnZoGHuRL4T6HomZnQP8BkgH/uru\n95jZnUCBu082s2HAJOAwYDew3t2PNLMTgEeI3YRPA37j7o8F++xJ7MZ9G2AucIW7H3IscT21Jcly\nd95cvpE/zljBuys/o1WTTK48oTvfPqE7bZrpCXRpmPRCYhwFidSmuR9v4k8zVzBt0afkZKZx2bBu\nfO+kHnQ5rGnUpYnUKgVJHAWJhKFowzYembmSSXPXAjB6SCeuPrkXfTu0iLgykdqhIImjIJEwrdu8\ni8feWsVTwTD2ZwzowDWn9OLoIw6LujSRpChI4ihIpC5s2hEbxv5v78SGsT+mRxuuOaUXp/TN1TD2\nUi8pSOIoSKQu7SwrZ8L7a/jLmytZt2U3/Q9vwTWn9OLcQRrGXuoXBUkcBYlEoay8gsnBMPZFG7bT\ntU0Txo7oxVeP7kJOpoaxl9SnIImjIJEoVVQ4rxZ+yh9mrGDems20a57Fd4b34IrjjqBVEw1jL6lL\nQRJHQSKpwN15b9Xn/HHGCmYuK6F5dgbnDe5Iu+bZNMvOoHlOBi2yM2gefG+enUGLnP8uZ2foLEbq\nVqJBklFVBxGpHWbGcT3bclzPtixat4U/zVzJ1IXr2V5azt4ExvLKTLe4kMmMhU5c0LTIzogFUnwo\nfSmQMmmWnU6zrAzS0nTzX2qPgkQkAkd2asX/Xj4UiJ2p7N5TwbbSPWzfXc6O0r1ffN9eGvts2/c9\nrm377nJKtpWyauOOYP0edu9JbCyw5gc78wnCKP5MKD6I9t8uK0MPD4iCRCRyZkaTrHSaZKXTPsl3\nGffsrWBHEDw7ymJhsy0+gL60vCcIpb1s372HT7fu/u/60nISueqdlZH2xZlPZvBEWvzlcj/gy5eH\n697X98tt8X39wLZK6qr0NxPZV6V9v/wD+/9eZfVXvuX+dR+678H+N6pq2wPr+3LDi9efRK/cA+YI\nrFUKEpEGJDM9jdZNs2jdNLnxv9ydnWV7vzgb2nHAmVEshOJDqnxv3F9gduDX+Hdp4i+s7Wv+clsl\nfb+0z7j1lW5/YN8vtX3pyt6h93Vg/8p/v5Iyg/UHv4yY3H4P/hvxS3XxQIeCREQOYGY0Cy5zdWgZ\ndTWS6nSBU0REkqIgERGRpChIREQkKQoSERFJioJERESSoiAREZGkKEhERCQpChIREUlKoxj918xK\ngNU13LwdsLEWy4lSQzmWhnIcoGNJVQ3lWJI9jiPcPbeqTo0iSJJhZgWJDKNcHzSUY2koxwE6llTV\nUI6lro5Dl7ZERCQpChIREUmKgqRqj0ZdQC1qKMfSUI4DdCypqqEcS50ch+6RiIhIUnRGIiIiSVGQ\nJMDM7jKz+WY2z8xeNrNOUddUU2b2kJktCY5nkpm1jrqmmjCzr5rZIjOrMLN6+XSNmY00s6VmVmRm\nt0RdT02Z2V/NbIOZLYy6lmSYWVczm25mhcG/Wz+MuqaaMrMcM3vfzD4MjuXnof6eLm1VzcxauvvW\n4Pv1QJ67Xx1xWTViZmcBr7t7uZk9AODu4yIuq9rMbABQATwC3OjuBRGXVC1mlg4sA84EioHZwOXu\nvjjSwmrAzEYA24F/uPvAqOupKTPrCHR09w/MrAUwBxhTT/8/MaCZu283s0zgLeCH7j4rjN/TGUkC\n9oVIoBmVTLdcX7j7y+5eHizOArpEWU9NuXuhuy+Nuo4kHAMUuftKdy8DJgAXRFxTjbj7G8DnUdeR\nLHf/xN0/CL5vAwqBztFWVTMesz1YzAw+of29pSBJkJndY2ZrgG8AP4u6nlryXWBq1EU0Up2BNXHL\nxdTTv7QaIjPrDgwF3ou2kpozs3QzmwdsAF5x99CORUESMLNXzWxhJZ8LANz9NnfvCjwBXBdttYdW\n1bEEfW4DyokdT0pK5DjqMaukrd6e6TYkZtYcmAj8aL+rEfWKu+919yHErjocY2ahXXbMCGvH9Y27\nn5Fg1yeBF4E7QiwnKVUdi5ldCZwHnO4pfJOsGv+f1EfFQNe45S7AuohqkUBwP2Ei8IS7Pxd1PbXB\n3Teb2QxgJBDKAxE6I0mAmfWJWxwNLImqlmSZ2UhgHDDa3XdGXU8jNhvoY2Y9zCwLuAyYHHFNjVpw\ng/oxoNDdfxV1Pckws9x9T2SaWRPgDEL8e0tPbSXAzCYC/Yg9JbQauNrd10ZbVc2YWRGQDXwWNM2q\nj0+gmdmFwP8CucBmYJ67nx1tVdVjZucAvwHSgb+6+z0Rl1QjZvYUcAqxkWY/Be5w98ciLaoGzOxE\n4E1gAbE/6wC3uvuU6KqqGTMbDPyd2L9bacDT7n5naL+nIBERkWTo0paIiCRFQSIiIklRkIiISFIU\nJCIikhQFiYiIJEVBIlILzGx71b0Ouf2zZtYz+N7czB4xsxXByK1vmNmxZpYVfNeLxJJSFCQiETOz\nI4F0d18ZNP2F2CCIfdz9SODbQLtgcMfXgEsjKVTkIBQkIrXIYh4KxgRbYGaXBu1pZvaH4AzjP2Y2\nxcwuCTb7BvBC0K8XcCxwu7tXAAQjBL8Y9H0+6C+SMnSKLFK7LgKGAEcRe9N7tpm9AQwHugODgPbE\nhij/a7DNcOCp4PuRxN7S33uQ/S8EhoVSuUgN6YxEpHadCDwVjLz6KTCT2F/8JwLPuHuFu68Hpsdt\n0xEoSWTnQcCUBRMviaQEBYlI7apsePhDtQPsAnKC74uAo8zsUH82s4HdNahNJBQKEpHa9QZwaTCp\nUC4wAnif2FSnFwf3SjoQG+Rwn0KgN4C7rwAKgJ8Ho9FiZn32zcFiZm2BEnffU1cHJFIVBYlI7ZoE\nzAc+BF4Hbg4uZU0kNgfJQmLzzL8HbAm2eZEvB8v3gMOBIjNbAPyZ/85VcipQ70ajlYZNo/+K1BEz\na+7u24OziveB4e6+PpgvYnqwfLCb7Pv28Rwwvp7PVy8NjJ7aEqk7/wkmG8oC7grOVHD3XWZ2B7E5\n2z8+2MbBBFjPK0Qk1eiMREREkqJ7JCIikhQFiYiIJEVBIiIiSVGQiIhIUhQkIiKSFAWJiIgk5f8D\nKvAxJUYAaMAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107b65b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# scores_：dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold,\n",
    "# Each dict value has shape (n_folds, len(Cs))\n",
    "Cs = [1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]\n",
    "n_Cs = len(Cs)\n",
    "n_classes = 9\n",
    "scores =  np.zeros((n_classes,n_Cs))\n",
    "\n",
    "for j in range(n_classes):\n",
    "        scores[j][:] = np.mean(lrcv_L1.scores_['Class_'+ str(j+1)],axis = 0)\n",
    "    \n",
    "mse_mean = -np.mean(scores, axis = 0)\n",
    "plt.plot(np.log10(Cs), mse_mean.reshape(n_Cs,1)) \n",
    "#plt(np.log10(reg.Cs)*np.ones(3), [0.28, 0.29, 0.30])\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('logloss')\n",
    "plt.show()\n",
    "\n",
    "#print ('C is:',lr_cv.C_)  #对多类分类问题，每个类别的分类器有一个C\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.32140535,  0.23499902,  0.16034243,  0.13795419,  0.13536477,\n",
       "        0.13543209,  0.13548341])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mse_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个score似乎和GridSearchCV得到的Score不一样:("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 0.135364772754\n"
     ]
    }
   ],
   "source": [
    "best_C = np.argmin(mse_mean)\n",
    "best_score = np.min(mse_mean)\n",
    "print Cs[best_C], best_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cPickle\n",
    "\n",
    "cPickle.dump(grid.best_estimator_, open(\"Otto_L2_tfidf.pkl\", 'wb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
